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ABSTRACT 

The use of formal written standards as a way to 
improve a product or service is discussed and illustrated by an 
analysis of the planned development of standards for educational data 
reporting. The Cooperative Educational Data Collection and Reporting 
Standards (CEDCARS) project is intended to apply to a wide range of 
educationa] data collection and reporting. Conducting a review of 
existing related standards was a logical first developmental step. 
The standards reviewed include the following: (1) National Center for 
Education Statistics "Standards and Policies" (1987); (2) Energy 
Information Administration "Standards Manual, U.S. Department of 
Energy, Office of Statistical Standards" (1989); (3) General 
Accounting Office "Government Auditing Standards" (1988); (4) Joint 
Committee on Standards for Educational Evaluation "Standards for 
Evaluation of Educational Programs, Projects, and Materials" (1981); 
(5) Evaluation Research Society "Standards for Evaluation Practice" 
(1981); (6) National Center for Health Statistics "Draft Standards on 
Statistics and Information on Effects of the Environment on Health" 
(1980); and (7) "Standards for Educational and Psychological Testing" 
(1985). Qualities required of the eventual CEDCARS standards, 
scheduled for draft completion in 1991, are reviewed. Twenty-two 
exhibits and a 66-item list of references are provided. (SLD) 
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The inability of the United States to compete in world markets as the productivity 
of our competitors increasingly outdistances our own is a signal that our nation's education system 
is in need of improvement. Jobs are requiring greater skills, and the demand for skilled workers 
continues to increase. At the same time, the national pool of educated and skilled people grows 
smaller while the number of potentially unemployed people rises. The recent educational 
performance goals set by President Bush and the governors are an expression of the strong interest 
in our society in improving the quality of the education system. 

This paper is not directly about standards for improving the quality of our 
educational performance, however, as professional statisticians and educational researchers, we 
believe that in order to guide and inform the people who will make the important policy choices to 
achieve educational performance goais, there is a clear need to improve the quality of the national, 
state, and local educational data systems. The antecedent to this data quality improvement, in our 
view, is the development of written standards for educational data. For this reason, the National 
Center for Education Statistics, as part of National Cooperative Educational Statistics System, 
established a project to develop standards for improvLng the quality of education data. 

In July 1989, the National Center for Education Statistics (NCES) contracted with 
Westat, Inc., of Rockville, Maryland, to facilitate and coordinate the development of a set of 
educational data reporting standards. The project was named "CEDCARS" for Cooperative 
Educational Data Collection and Reporting Standards. While ihe standards were to have a 
primary focus on the educational data reported by States and school districts to the Federal 
government, they were also intended to be applicable to a wide range of other educational data 
collection and reporting. The standards were to be developed cooperatively by a Task Force 
drawn from the intended user groups, rather than by the Federal government. 

The use of formal written standards as a way to improve product or service 
(operational) qualify has become increasingly prevalent, and the American National Standards 
Institute (ANSI) currently has over 10,000 officiaUy approved standards. Within the area of 
statistical data coUection and analysis, formal written standards are relatively recent, but are 
increasingly seen as a way to enhance the accuracy, comparability and utility of statistical data 
upon which aU types of education decisions are made (e.g. policy, administration, classroom, etc.). 

Conducting a review of related standards was seen as the logical first task for the 
CEDCARS project. We were concerned not only with the content of the standards, but also with 
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the process that had been followed in arriving at them, the format used, and the subsequent field 
testing, dissemination and use of the sundards. This paper summarizes the results of this review, 
presents a brief description of the process being foUowed in the drafting of the CEDCARS 
standards, and discusses the Total Data Quality Improvement concept as a mode?, underlying the 
development of the CEDCAR sundards. It is a condensed version of a longer in-depth standards 
review prepared for the project. The paper is presented in five sections: (1) The Definition of 
Standards and the Qualities of Good Standards; (2) The Standards Setting Process; (3) A 
Comparison of Selected Standards Dimensions; (4) The Topics of the Standards; and (5) 
Summary of CEDCARS Project and discussion of the Total Data Quality Improvement model. 

In all, 16 sets of professional standards from a variety of sources were reviewed, 
and 7 were selected for indepth consideration. These included four sc )f standards on statistical 
data coUection from Federal agencies in the diverse areas of education energy, accounting, and 
health, and three sets of standards from the professional educational research community. The 
foUowing standards were reviewed in depth: 

■ National Center for Education Statistics (NCES), Standards and Policies ( 1987) 

■ Energy Information Administration (EIA), Standards Manualy U,S. Department of 
Energy, Office of Statistical Standards (1989) 

■ General Accounting Office (GAO), Government Auditing Standards ( 1988) 

■ National Center for Health Statistics (NCHS),'/)ro/ir Standards on Statistics and 
Information on Effects of the Environment on Health (1980) 

■ Joint Committee on Standards for Educational Evaluation, Standards for 
Evaluation of Educational Programs, Projects, and Materials (1981) 

■ Evaluation Research Society (ERS), Standards for Evaluation ftactice (1981) 

■ American Educational Research Association (AERA), American Psychological 
Association (APA \ and National Council of Measurement in Education (NCME), 
Standards for Edu^Mtional and Psychological Testing ( 1985) 

The proposed CEDCARS standards embody characteristics of both government 
and professional standards, but are unique in that the}' do not fall entirely into either category. 
Although the development effort is being sponsored and reviewed by the National Center for 
Education Statistics, the Task Force, comprised primarily of members outside the Center, has 
responsibility for the development of the content of the standards. 



1. The Dennition of Sundards and the Qualities of Good SUndards 



Among.ihe first questions we asked were, "How have the previous groups viewed 
standards?" "What do they hope to attain by having standards?" and "What are the essential 
qualities of good standards?" 

Of the seven sets of standards, only those written by the Joint Committee on 
Standards for Educational Evaluation include a formal definition of the term; however, several of 
the standards-setting groups have described the intended nature or characteristics of their 
standards. From these descriptions, it is possible to infer the way in which standards are 
conceptualized by the group. 

The Joint Committee defined a standard as "A principle commonly agreed to by 
experts in the conduct and use of evaluation for the measure of the value or quality of an 
evaluation" (Joint Committee, 1980). 

The ERS society has noted that their standards take the form of "simple 
admonitory statements" (ERS, 1982). 

Both EIA and NCES refer to standards in very similar ways. EIA distinguishes 
between "orders" and "standards," and refers to both in their manual. In the introduction, EIA 
states that "standards outline the fundamentals of data quality and production." Orders address a 
variety of issues and "provide specific procedures, detailing the interrelationships of EIA units in 
the processes being addressed." EIA states that the purpose of the Standards Manual is threefold: 
( 1) to increase the reliability and validity of EIA d^.a and modeling systems; (2) avoid duplication 
and promote efficiency and economy by providing tor interchangability and transferability of 
systems and procedures; and (3) remove ambiguities and inconsistencies from EIA products and 
provide the clearest possible data presentation (EIA, 1989). 
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NCES uses very similar language in the foreword to their standards, which were in 
part modeled on those of the EIA. NCES states: 

They represent the minimum level of quality we would expect in any of our efforts 
or those of our contractors and grantees. For the Center they provide a means of 
and assurance of consistency between and within the studies the Center conducts. 
Finally, users of Center data have before them clear documentation of methods 
and principles the Center employs in the collection of data. (NCES. 1987) 

Four elements predominate in these descriptions. As we pursued our own thoughts on standards, 
we focused on the following: 

L The role of stone A/ds in promoting increased levels of quality/best practice/vaiidity; 

2. The role of standards in promoting consistency, and ensuring that a certain 
expec* nciesaremet; 

3. The role of standards as allowing evaluation and fostering documentation both as a 
means of improving quality and as a means of allowing independent review and use; 
and 

4. Standards as reflecting a consensus. 
Standards as Rules or as Models 

Those who have discussed the concept of standards (Cordray, 1982) have noted 
that a standard can be differentially viewed xs minimum criteria and/or rules, or as modeb of 
desired behavior. Standards used as rules or minimum criteria tend to be more specific and are 
more frequently characteristic of accrediting groups and those with authority to require 
compliance. Used as a model, a standard provides a principle or ideal - something to which we 
aspire. Standards as models are often less specific than standards intended as rules and carry less 
authority to enforce compliance. CEDCARS standards as they are currently being developed are 
viewed primarily as models of desired behavior rather than rules requiring (X)mpliance. 



Qualities of the Standards Themselves 

In reviewing lists of the qualities of "good standards" (Chalk, Frankel, and Chafer, 
1980; APA/AERA/NCME Standards for Psychological and Educational Testing, 1985). and in our 
own thinking on the matter, we find that there is a certain dialeaic manifest. Often seemingly 
opposite elements are pui forth as important qualities. Some examples are as follows: 

■ Standards are to be useful in giving guidance to specific problems, yet they must have a 
wide applicability, 

m Standards must be easily understood in clear language, yet they must manifest a high 
level of technical sophistication, 

m Standards must be acceptable to all affected^ yet they must not pull the level of quality 
down for those already operating at a higher levels, 

m Standards must reflect the minimum best practice that all should achieve^ yet they 
should also be a force for upgrading, 

3 Standards should have u strong imperative and be internally consistent^ yet they must 
allow for the fact that they can not always be uniformly applied. 

■ Standards should promote consistency and comparability, yet they should not inhibit 
innovation. 

■ Standards should require that sufficient documentation be made available so that the 
adequacy of the project can be evaluated, yet they should not lead to more concern v ith 
documentation than actual quality (shadow compliance), 

m Standards should be comprehensive and cover each stage and aspect of the operation 
equally welly yet standards should allow for priorities to be established 

The CEDCARS project continues to struggle with achieving successful integration of these 
qualities. 

2. The Standards-Setting Process 

Of crucial interest to us is the question of how to set the standards. This question 
can be approached both in terms of the organizational structure followed (who sets the standards 
and how they are organized) and in terms of the actual steps taken to arrive at specific formulation 
of the words of the standards. We begin with a discussion of the organization of the process, and 
continue with discussing specific standard drafting/setting techniques. 



Tht Initiation of the Standards-setting Process 



Observers of the standards-setting process have identified some of the conditions 
which often accompany the initiation of a standards-setting effort. Based on a study of standards 
in accounting, auditing, and education evaluation, Jeri Nowakowski (1982) found the following 
conditions critical to establishing a compelling standards-setting rationale and procedure: 

Readiness - there was general agreement about the objectives and functions of the 
activity of concern. 

Capability - multiple relevant organizations would support the stand^^ds-setting 
process and were able to work together to assure participation of the profession. 

Need - performance was of uneven quality; c^jtside standards were being enforced 
or outside criticism was being made; there was agreement that sound practice could 
be differentiated from substandard. 

Feasibility the group could work with others in the area to achieve acceptance of 
the standards. 



These are similar to the conditions needed for achievements in other areas. For 
example, Neil Armstrong and others have identified four factors as crucial to the success of the 
manned space program to the moon. These were technological base (readiness), uneasiness about 
position in the world (need), catalytic event (capability), and a leader who focuses attention and 
makes the resources available (feasibility). 

An examination of the standards-setting effort of each of the seven sets included in 
the detailed review illustrates the presence of these conditions in each of the efforts, although in 
very different ways. Of the four sets of standards written by government agencies, two were done 
in direct response to a legislative directive. The other three sets of standards were developed by 
groups representing professions, and the history and organizational process foUowed in producing 
them reflects this basic difference. Exhibit 1 presents a comparative summary description of the 
origin and organization of the standards-setting effort of each of the seven standards. 
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Exhibit 1-a. Comparison of the organization and procedures for standard setting: 

government agency standards 



. Draft Guidelines for 

Standards and Policies Energy Information Government » Statistics and Information 

OrganiutioQ & for National Center Administration Auditing Effeas of the Envuonment 

Procedures for Education Stucistics Standards Manual Standards on Healrh 



MotivatioD 
for Initiating 
Process 



Recommended by the 
Center's Advisory Counal 
on Education Statistics 
(ACES) 



Increase validity/ 
reliability 
Avoid duplication 
Remove ambiguities 
Inconsistency 



Reliance on auditors has 
enhanced the need for 
standards to guide auditors 
and allow others to rety on 
auditors* worlc 



Health standards were being 
challenged, primarily because of 
the madequacy of saentific and 
medical evidence of the 
relationship between 
environmental conditions and 
presumed deleterious health 
effeas 



Intended 
Users 



NCES staff and contraaors 



Authority 



Validation 
Procedures 



Provision 
foi Revision 



Impieroenta* 

tion/Technical 
Assistance 



Required by NCES staff 
and contraaors 



Sent to staff and contraaors 
for fmal review 



Formal evaluation of the 
standards program to ensure 
that the standards have 
been implemented in all 
phases of the Center's 
work, and to review their 
operational feasibility 

The entire staff was 
involved in the produaion 
of the standards and wis 

thus introduced to the 
standards 



EIA Program 

Offices 

Contraaors 



Auditors and audit 
organizations 



Required Quality 
Control Officers 
responsible 



Staff review of 
standards 



There is a 
computerized 
on*line edition 



OfTice of Statistical 
Standards momtors 
compliance through 
system of ongomg 
reviews and periodic 
assessments/assistanc 
e also given for 
implementation 



Auditors and audit 
organizations must follow 
standards when required by 
law. regulation, agreement, 
rontraa, or policy 

Soliatation of comments 
and suggestions; reviewed 
by the Auditing Standards 
Advisory Council 

Not specified 



Formal system for issumg 
auditmg standards and 
related interpretations and 
guidance has been 
established 



Executive departments 
responsible for the 
administration of laws relating 
to the proteaion of the public 
health and safety or the 
environment 

Executive order requu'es each 
executive department to comply 
with requirements of the 
guidelines respeauig specific 
statistics 

Drafts reviewed by members 
from various agenaes, 
comments soliated from 
general public 

Legal provuion allows review 
and revision at least every three 
years 



Not specified 
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Organtut^oD Sc 
Procedures 



ELihibil 1-a. Comparison of the organization and procedures for standard setting: 
government agency standards (continued) 



Standirds and Policies 

for National Center 
for Education Statistia 



Energy Information 

Administration 
Standards Manual 



Government 
Auditing 
Standards 



Draft Guidelines for 
Statistics and Information 
Effeas of the Environment 
on Health 



Who Is National Center for 

Involved in Education Statistics 
Producing/ (NCES). Office of 
Sponsoring Educational Research and 

Improvement (OERI). U.S. 
Department of Education 



Energy Information 
Administration, 
OfTice of Statistical 
Standards 



General Accounting Office 



National Center for Health 
Statistia; OfTice of Health 
Research, Statistics and 
Technology; Pi*blic Health 
Scnace; U.S. Department of 
Health and Human Service 



• How is 
Standard* 
Setting Body 
Organized 



• Imtially drafted under 
contraa 

- Staff concerned with 
statistical data analysis or 
processing issues 
reviewed first drafts for 
many of the standards 

- Each of the Divisions 
discussed changes to the 
standards and made 
recommendations on 
additional standards 
needed 



Staff Admmistrator 
a^roved 



9«member drafting 
committee 

Mail*review panel 



Subcommittee on 
Environmental Health 
Statistia of the National 
Conunittee on Vital and 
Health Statistia helped 
formulate guidelines and 
review preparatory draits 

Individuals from several 
government agenoes 
reviewed preparatory drafts 

Members of the general 
public provided comments 
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Exhibit 1-b. Comparison of the organization and procedures for standard setting: 
professional educational research standards 



Sttndards for 

EvaJutiions of Standards for 

OrgaDizatioo & Educational Programi» Educational and Standards for 

Procedum Projeas and Matmals Psychological Testing Evaluation Praaice 



■ MotivatkNi 
for Initiatiof 
Process 



Evolved from the 1974 
revision of the Standards for 
Educational and Psychological 
Tests (Amencan Psychological 
Association), which 
recommended a compy 
volume focused on the 
evaluation of educational 
programs, projeas and 
materiab 



To improve the quality of 
appropriate professional 
aatvities; ERS cvaluaton 
perform a great many kinds 
of evaluative aatvities 
besides those covered by 
GAO and by the Joint 
Committee 



The 1974 Standards 
publication was becoming 
outdated because of new 
problems and issues 
LHvotving testmg 



• Intended People who commission. 

Users condua or employ results 

of evaluations to improve 
education: teachers, 
adnunistrators, evaluators, 
curriculum specialists, school 
board members, legislators, 
counselors, leaden of 
educational assocutions and 
parents 



■ Authority 

• Validattoo 
Procedures 



Voluntary 

Field tests, detailed critiques 



People engaged m program 
evaluation 



Voluntary 
Mail-review panel 



Professional test developers, 
sponsors, publishers and 
users 



Voluntary 

Drafts reviewed by members 
of the sponsoring 
organizations 



Provisioo for 
Revision 



Users complete feedback 
forms as part of ongoing 
review and revision process 



Penodic reexamination and 
revision 



Contmual need 
acknowledged for 
monitoring and revising 
document 



Implementatioa/ 

Technical 

Assistance 



Information packet for users Not specified 



Not specified 
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Exhibit 1-b. Comparison of the organization and procedures for standard setting: 
professional educational research standards (continued) 



Procedures 



Standkrds for 
Evaluations of 
Educational ^"gnmi, 
Projects and Materials 



Standards for 
Educational and 
Psychological Tcsiing 



Standards for 
Evaluation Praaice 



Wbo Is In?olf«d 

ii Producinc/ 
Spoosorinf 



Joint Comnuttec on Standards 
for Educational Evaluation, 
initialiy appointed by the 
American Educatioiial 
Research Association 
(AERA), the American 
Psychological Association 
(APA), and the National 
Council on Measurement m 
Education (NCME) 



Evaluation Research Sooety 



American Educational 
Research Assoaation 
(AERA)» American 
Psychological Association 
(APA) and the National 
Council on Measurement in 
Education fNCME) 



• How ;s Standard- • Projea staff at Western 
Setting Body Michigan University 

Organiied coordmated projea 

aaivities 



9*mtmber drafting - 12 members on a 

committee committee appomted by 

the 3 sponsoring 
Mail*rcview panel organizations 



• Wational panel of 29 
evaluation experts helped 
draft the initial set of 
standards 

• 16 graduate students at 
Western Michigan 
University reviewed and 
assisted in revising the 
standards 



• 125 people served as 
adv-^ers to the committee 

• Ad hoc committees 
representing the 
governing bodies of the 
sponsoring organizations 



National review panel of 42 
educaton and sodal 
sdentiiu reviewed drafts of 
project repofts 

21> evaluators and 
evaluation teams field- 
tested the undards 



• M-^ ptofAt participated 
in national hearings on the 
standards 
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Who Is Involved in Sponsoring and Producing the Standards? 

In each of the seven cases we examined, the standuds-setting activity was sponsored by 
a group that was either representative of the intended users, or in the case of the government standards, 
had authority over the intended users. The sponsors of the government standards were acting in 
response to an outside directive or recommendation. The professional standards were 
recommended by a group representing practitioners who then became the sponsors. In most cases, 
the work of producing the standards involved a task force or committee working either in a 
drafting or review capacity. In some cases this work was included as pan of an existing sub* 
committee (NCHS, GAO). In others, a task force or joint committee was formed which later 
dissolved or changed focus after the standards were drafted (ERS; AERA/APA/NCME). In the 
case of the Joint Committee on Standards for Educational Evaluation, the standards-setting body 
became its own organization with a budget and ongoing projects. ELA has an Office of Statistical 
Standards which also monitors compliance with the standards. 

How Is the Standards-setting Body Organized? jt both the government and 
professional group standards, the organization of the standards-setting process usuaUy involved at 
least nvo separate components ~ a drafting committee and a review committee - and sometimes a 
third component - an advisory panel. The Joint Committee standards-setting body also included 
an evaluation team to field test the standards and a validation panel to monitor and evaluate the 
field lest. 

Drafting committee members ranged from concerned staff to national experts to 
individuals appointed by sponsoring organizations. Some worked entirely as a committee in 
drafting the standards, either through intensive group meetings or dividing the writing into 
sections among the members. Others had one or two individuals who drafted some or aU of the 
standards, which the working committee then revised. 

Each of the standards-setting efforts involved some form of outside review by those 
interested persons who were not directly involved in drafting the standards. The review 
committees on the various standards projects were organized in different forms - from mail panels 
to public hearings. 

The CEDCARS standards-setting body has a 15 member Task Force composed of 
representatives of the prospective user groups and a 32 member Task Group composed of 
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members of the Task Force and experts in the applicable data collection and analysis fields. The 
Task Force serves as the decision making body for the project. The Task Group has had the 
responsibiL'tj' for initially drafting the standards according to the specifications developed by the 
Task Force. The Task Force participated in a delphi like process in arriving at the specifications 
for the standards. Westat, as contractors for the project, has the responsibility for coordination of 
the project and editing the standards into a usable document. The standards will be reviewed by 
each of the anticipated major user groups and field tested on several actual data coUection and 
analysis efforts representing the differing kinds of data collection/reporting. 

The project is being conducted in three phases: The first phase involved deciding 
on the specifications for the standards; the second phase, in which we are currently involved, is the 
actual drafting of the standards; and the third phase is review and field testing. 

What Was the Motivation for Initiating Process? The original preparation of any 
set of standards typically occurs in response to perceived deficiencies or in an effort to avoid 
deficiencies in current practices in the field (EIA, NCES, ERS, GAO, Joint Committee). Each of 
the standards derived its legitimation front being recommended or required by an organization 
with a recognized interest or obligation to improve the quality of information or professional 
practice. In some cases this recommendation was sought by those interested in seeing that 
standards were drafted, and the group subsequently became the standards-setting group. 
Revisions to standards usually are recommended because the initial standards are becoming 
outdated due to new problems and issues in the field (AERA/APA/NCME, NCHS, EIA, and 
GAO). 

Who Are the Intended Users? Deciding for whom the standards are intended is 
essaatial for determining the topics, content, and language of the standards. Users of the four sets 
of government standards were determined by legislative or executive fiat. For example, GAO 
Standards are intended for fmancial and performance auditors of all government funded 
programs; NCES and EIA Standards, for use by all staff and contractors; and NCHS Standards, for 
executive departments responsible fcr the administration of laws relating to the protection of the 
public health and safety or the environment, and also for the general public. 
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Vfftr grcups for the standards drafted by organizations representing professional 
groups were determined by virtue of common field of endeavor. AERA/APA/NCME Standards 
are for professional test developers, sponsors, publishers and users; ERS Standards are for people 
engaged ir program evaiu ution. The Joint Committee Standards are intended for teachers, 
administrators, evaluators, curriculum specialists, school board members, legislators, counselors, 
and educational association ler iers, as weU as for parents. 

In general the broader the audience of intended users and scope of the standards, the 
more general the formulation of the standards. Standards that attempt to be applicable to a wide 
range of endeavors must sacrifice some level of specificity. Similarly, those that are intended to be 
of use to the general public must use language accessible to people interested in the field, yet 
lacking the expertise of trained professionals. We discuss CEDCARS users in section 5. 

What Is the Authority of the Standards? Compliance with some standards is 
required by law, employment agreement, or contract. Other standards are recommended for the 
user's professional improvement, but carry no legal authority. The GAO, NCES, EIA, and NCHS 
standards are examples of the mandated variety; the Joint Committee, ERS, and 
AERA/APA/NCME standards are the recommended type. The authority the standards carry will 
affect their content and wording. 

The CEDCAR Standards are intended to be voluntary. However, the suggestion 
that they be used to conduct audits has already arisen. As voluntary standards they can present a 
model of best praaice, rather than a minimum that can be expected. If, however, they are to be 
used to conduct audits, they would have to be drafted in a manner that was more specific to 
individual data coUections. It has also been suggested that the focus be on evaluating the processes 
followed rather that the actual audit of the data. 

What Were the Validation Procedures? To validate the standards in terms of their 
effectiveness, ethicalncss and appropriateness, each of the standards we reviewed was subjected to 
one or more of the foUowing: field tests, professional critiques, field reviews of drafts and 
solicitations of comments from the general public. In deciding who will review the standards, it is 
important to consider the intended users. All groups for whom the standards are intended should 
contribute to the review. CEDCARS validation procedures are currently in the development 
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phase. In addition to extensive reviews, we are in the process of recruiting school districts and 
studies to field test the standards on their own data coUection/reporting. 

What Is the Provisioo for Revision? Most standards-setting bodies employ 
language that asserts that the standards must be dynamic and cannot be viewed as complete and 
final. Each of the drafts of standards we reviewed also acknowledged the need for revision. They 
differed in the degree of specificity of the revision plan provided and in the formality of the 
revision procedures. The GAO Government Auditing Standards have been revised several times, 
as have the APA/AERA/NCME Psychological and Educational Testing Standards. These 
revisions appear to have occurred when a significant body of persons involved in uae organization 
determined that it was time for revision. The NCHS Standards legislation calls for revision every 
three years. The NCES Standards, initially drafted in 1987, are currently being reviewed. The 
EIA Standards have had at least one revision. The Joint Committee on Standards for Educational 
Evaluation asks users to complete forms reporting on their application of the standards. This 
feedback is incorporated into the review and revision process. 



Implementation* Procedures for implementation and technical assistance not only 
aid the users, but also provi je a means for the standards setters to obtain critical feedback. The 
Joint Committee provides information packets to all interested users, and NCES involved its 
entire staff in the standards production, both as a means to familiarize them with the standards 
and to assist them in the later use. GAO Standards includes a summary format system for helping 
users implement the standards. Unfortunately, while considerable information !5 available on the 
standards-setting process, there is much less information on the implementation of standards. 



Specific Ways of Arriving at the Standards 

To this point, our discussion has focused on the organization of the standards- 
setting process. The other aspect of the endeavor relates to how the standards-setting body 
actually determined what the standards should be. It is helpful to consider the components which 
may go into arriving at a set of standards. 
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In the context of discussing setting standards in high risk areas such as regulating 
the nuclear industry. Fischhoff ( 1983) discusses four means of arriving at standards. These are 
summarized in Exhibit 2, which was taken from his repo . The first cf these is fomial analysis. 
This involves a formalized process in which a standard or set of standards that offers the group the 
highest possible benefit or a model of the policies and procedures yielding highest quality is 
drafted. This is determined by estimating the consequences following from implementation of each 
possible standard This might involve applying selected principles to an effort. This approach has 
the advantage of being systematic, but it may be impractical and over-centralize power either to an 
individual or a particular concept. 

A second method, professional Judgment, if probably the most frequently used. 
The professional technical community that creates technologies has always been the primary 
source of standards governing themselves. These groups presumably imow the most about their 
professional endeavors. This method has the advantage of being realistic, but in some areas, 
especially those involving risk, issues of vested interest became relevant. Studies from the area ol 
testing standards have shown that providing the group of experts with performance data prevents 
adoption of unrealistic standards (Norcini, Shea, and Kanya, 1988). This fmding may serve to 
point out the importance of our understanding the current practices within the areas of education 
data collection and reporting before setting the standards. 

A third means is that of political processes, which includes the involvement of 
informed lay persons. The credibility of a particular political process depends upon how 
representative it is, how well it exploits technical knowledge, how well its lay participants resist 
bullying by expert opinion, and how well-informed lay persons are. The fourth method is that of 
revealed prererences. In this method standards are adopted which emerge from actual practice. 
TYiQ locus of wisdom is the norm of past social processes. A step in this process might be a survey 
of current practices. 
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Exhibit 2. Methods of setting standards 
(from Standard Setting Standards by Baxuch Fischiioff» 
Oak Ridge National Laboratory, 1983) 



tjOCUt of 

Wkdom 



Deicnptw 



PotcntuI 
Advintigct 



Potential 



Formal A/ulvnt 



Profeuional 
Judgment 

Politicil Procettci 



Revelled 
Preferences 



FonntUzcd 
intelkcnial 
proccttct 



Intuitive 
incellecnul 
pr 



Body pobttc 



Past social 
processes 



ChooM standard 
offenr^ highctc 
utility (or best 
cost-benefit 
trade ofT) 

Let technical 
expcru identify 
best standard 



Have lay poups 
set sundards, 

informed by 
technical advice 

Adopt sundard 
impi Kitty 

emerpni in actual 

decisions 



Systematic explsai 
sophisticated 
techniques 



Realistic 
implenKntable 
creative 
comproreisci 

Broad penpectivc 
ieptunAcy 
open to cntmsm 



Refkco deeds 
shaped through 

expenenct 
influenced by 

whole soaety 



Impractical 
oversold 

centralues power 



Vested interesu 
incomplete 

perspectives 
instrucuble 



Uninformed 

unrealistic 

unstable 



Inefficient 
unfair 

insensitive to nsk 
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Few standards-setting processes seem to be conducted following only one approach. 
Rather, most involve combinations of each. The CEDCARS project, as currently organized, also 
combines these approaches in the standards development process. The Task Force and Task 
Groups represent the perspective of professionals and also contribute knowledge of what is 
preferred current practice. The review process will involve political and audience concerns and the 
field test will show the extent to which the standards reflect actual current practice. We are also 
doing a formal analysis of the data collection and analysis process from a management perspective 
and present some of our thinking on this in section S. 

The American National Standards Institute (ANSI) Procedures 

The American National Standards Institute has put forth requirements for 
standards approval that deal almost exclusively with the process that is followed in drafting and 
reviewing the standards. ANSI approval of standards is intended to verify that the principles of 
openness and due process have been followed in the approval procedure and that a consensus of 
those directly and materially affected by the standards has been achieved (American National 
Standards Institute, Procedures for the Development and Coordination of American National 
Standards, 1987, foreword). 

In order to achieve ANSI approval, standard setting must meet certain due process 
requirements. Due process means that any person (organization, company, government, agency, 
etc.) with a direct and material interest has a right to participate by expressing a position and its 
basis, having that position considered, and appealing if adversely affected. Participation is to be 
open to all persons directly or materially affected by the activity. The standards development 
process should have a balance of interests and shall not be dominated by any single interest 
category. Interest categories include the producer of the standards, the users, and those having a 
general interest in the process. Appropriate representation is to be sought for each of these 
groups. 

In order to obtain ANSI approval, the standards developer has to be accredited by 
ANSI, and there must be evidence that the due process requirements were met and that consensus 
was achieved. The standards must be within the scope previously registered with ANSI, conflicts 
with other standards resolved, other known standards examined and duplication avoided, appeals 
completed, and ANSI patent policy must be met. 
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A potential standard developer may hi accredited to use one or more of the 
recognized methods of developing evidence of consensus-Accredited Organization Method, 
Accredited Standards Committee Method, and the Accredited Sponsor using the Canvass Method. 
While none of the seven standards reviewed in detail in this report currently have American 
National Standards Institute approval, the Joint Committee on Standards for Educational 
Evaluation has recently received accreditation from the group as a Standards setting Committee. 
They will be submiaing their Standards for Evaluation of Education Programs, Projects, and 
Materials for ANSI approval. 

3. A Comparison of Selected Standards Dimensions 

In preparing to draft standards several decisions about the specifications for the 
standards had to be made. During Phase I, we used the review of related standards to generate 
the options concerning format and other dimensions of the proposed CEDCARS standards. Task 
force members then reached concensus concerning these dimensions. 

Standard Organizing Concepts and Corresponding Standard Groupings 

Most sets of standards of any length are organized into groups according to some 
explicit or implicit framework. Exhibit 3 presents the standard groupings (major sections into 
which the standards are organized) for each of the seven sets of standards. In looking at the seven 
sets we have examples of three types of conceptual organization: 

■ Ainctional temporal 

■ hinctional temporal combined with sections on speciflc types of applications, and 

■ theoretical conceptuaL 
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Exhibit 3. 

Standard spedficatioD element: SUndard Groupings, Focus and Date of Each of the Seven Sets of Standards* 



NaUonal Center for EducaUon SUtisUcs (NCES) 
Standards and Policies 

Focus; All surveys conducted by Center staff and 

contractors 
Date: 1987 

Sta&dard organization: functional temporal 

Major Sections: 

■ Standards on Planning 

■ Standards on RFP Development/Contract 
iVfonitoring 

• Standards on Implementation/Data 
Collection/Processing 

■ Standards on Release/Publication of Date 

■ Standards on Sampling and Non-Sampling 
Error 



Eoen{y lofonnation Systems 
Standards Manual 

Focus: Statistical Data on energy 
Date: 1)85 revised 1989 

Standard organization: functional temporal with scciions on 
speciTtc types of applications 

Major Sections: 

■ EIA Model and Computer Language Orders 
and standards 

■ Interagency Standards 

■ Data Systems Standards 

■ Data Collecting Standards 

■ Data Presentation Standards 



General Accounting Office 
Government Auditing Standards 

Focus: Audits of governmental organizations, 

programs, activities, and functions 
Date: 1980 

Standard Organization; functional temporal with 
sections on specific types of applications 

Major Sections: 

Introduction 

Types of Government Audits 
General Standards 

Field Worlc Standards for Fmandal Audits 
Reporting Standards for Fmandal Audits 
Field Worlc Standards for Performance 
Audits 

Reporting Standards for Performance 
Audits 



National CenUr for Health SUtistics 
Draft Guidelines for SUtistics and 
infonnatioD on Effiects of the 
Environment on Health 

Focus: Statistics for determining effects 
of conditions of employment 
and environment on public 
health 

Date: 1980 

Standard organization: functional 
temporal 

Major Sections: 

■ Guidelines for Collection 

■ Guidelines for Compilation 

■ Guidelines for Analysis 

■ Guidelines for Publication 
• Guidelines for Distribution 



^Refers to major sections into which the standards are organized; related to organizing concepts 
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Exhibit 3. 



Standard spcdficttion clement: Slaodard Groupinp, Focus and Date of Each of the Seven Sets of Standards* 

(continued) 



Joln^ Comnittcc oa Standards for Educational 
Evaluation 

Standards for Evaluaeioos of Educational 
Progranu, Projects, and Materials 

Foois: Education evaluation 
Date: 1981 

Standard Organization: theortical conceptual 

Major Sections: 

■ Utility Standards 

■ Feasibility Standards 

■ Propriety Standards 

■ Accuracy Standards 

Functional Organization: 
Administering 
Analyzing Information 
Budgeting Evaluation 
Deciding Whether to Evaluate 
Defining Evaluation Problem 
Collecting Information 
Contracting Evaluation 
Reporting Evaluation 
Sta£Rng Evaluation 



Evaluation Research Society 
Standards for Profraoi Evaluation 

Focus: Program evaluation 
Date: 1982 

Type or organization* functional temporal 

Major Seaions: 

■ Formulation and Negotiations 

■ Structure and Design 

a Data Collection and Preparation 

■ Comunication and Disclosures 

■ Utilization 



Joint Committee of AERA/APA/NCME 
SUndards for Educational and Psychological 
Testing 

Focus: Educational and psychological tests 
Date: 1985 

Standard organization: functional temporal 
combined with sections on spcdfidc types of 
applications 

Major Sections: 

■ Technical Standards for Test Construction 
and Evaluation 

■ Professional Standards for Test-Use 

■ Standards for Particular Applications 

■ Standards for Administrative Procedures 



^Refers to major sections into which the standards arc organized; related to organizing concepts 
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ERS Standards use what we are calling a functional temporal order. In this type of 
organize aon che standards are organized into groups reflecting the steps one takes in planning, 
collecting, analyzing, and reporting information. For example, the ERS Standards arc organized 
into five major sections: formulation and negotiation; structure and design; data collection and 
preparation; communication and disclosure; and utilization. This grouping of standards represents 
the sequential ordering of the major steps in planning and conduaing studies. It has the advantage 
of being straightforward and easily understood. It also presents a framework in which 
comprehensiveness can be attempted m a systematic manner. 

The NCES Standards and Policies and the NCHS Draft Guidelines for Statistia and 
hformation on Effects of the Environment on Health also generally follow a temporal functional 
ordering. Within this ordering, however, they include some standards or parts of standards that 
deal with specific types of studies (e.g., longitudinal studies or use of educational testing). These 
are not presented in a separate section. 

The sectional grouping of the GAO Government Auditing Standards, the AERA/ 
APA/NCME Standards on Educational and Psychological Testing, and the Energy Information 
Administration Standards Manual combine a functional temporal ordering with sections on 
particular applications. This type of organization is appropriate when the scope of the standards 
covers several types of unique endeavors that need sections dealing with the particular 
applications. For example, the GAO standards have separate sections dealing with financial and 
performance audits. 

The Joint Committee Standards for Evaluations of Educational Programs, Projects, 
and Materials and the second group of standards developed by the Joint Committee, those on 
evaluation of educational personnel, use a theoretical conceptual model for organizing the 
standards. These standards are organized into sections on utility, feasibility, propriety, and 
accuracy. 

The Joint Committee identified these concepts as corresponding to the four main 
concerns about evaluation. In introducing these concepts the Joint Committee notes that. The 
Committee was satisfied that standards which shape an evaluation so that it has these four 
characteristics are necessary and sufficient for sound evaluation in education" (1980, p. 13). Using 
this type of conceptual organization enables one to ensure that key concerns or qualities of a good 
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evaluation or data collection are manifest in th* standards. This typ>e of organization can give 
depth to the standards* 



The CEDCARS standards will foUow a temporal functional ord^r» The concensus 
of the Task Forr-^ was that this would be most easy to develop and use. 

Number and Length oT Standards 

Exhibit 4 summarizes the length and number of admonitory statements of each of the seven sets of 
standards. As can be seen the number of "standards" ranges from 21 for the NCHS Draft 
Guidelines for Statistics and Information on Effects of the Environment on Health and the NCES 
Standards and Policies to 195 for the AERA/APA/NCME Standards for Educational and 
Psychological Testing, However, a simple look at the number of formal standards can be 
misleading in comparing standards. Each set has different formats and several contain a large 
number of admonitory statements in cne standard. For example, the 21 NCES Standards contain 
about 300 admonitory statements, and the Joint Committee*s 30 Standards and the ElA's 24 
standards each contain about 340 admonitory statements. The very brief six-page Ev^luauon 
Research Society Standards for Evaluation Practice has 55 standards and also 55 admonitory 
statements. The AERA/APA/NCME Testing Standards have about the same number of 
standards as admonitory statements, since each admonitory statement is also a separate standard. 
The GAO Standards for Government Auditing has 22 statements that are labeled as standards; 
however, there are about 257 admonitory substatements numbered in the book of standards. 

One can also compare the standards on length of the document. The Joint 
Committee Standards, althou^ containing very short standard statements, include an illustrative 
case for each standard and subsequently are 155 pages long. The NCES standards contain almost 
as many admonitory statements, but fill only 38 pages. The proposed CEDCARS standards will 
consist of about 24 standards each with several guidelines and some with checklists. 

Components/Fomiat of Each Standard 

One of the most important decisions to be made in designing the specifications for 
the standards is to determine the components or format of each standard. Exhibit 5 summarizes 
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Exhibit 4. 

Standard specification element: Number of Admonitory Sutements* 



Standard set 



Number of 
separate 

Numbe. of standards gmdelines 
or "should" 
statements 



N^umbcr 
of 
pages 



NCES, SUndards and Policial 



21 



157 * 

145 points 38 
on 6 

checklists 



EIA» Energy Information 
Administration Standards 



24 + 
3 orders 



about 
340 



93 



GAO, Standards for Govemmerit Auditing 



22 



257 



76 



NCHS, Draft Guidelines for 
Statistics and Information on 
Effects of Environment on Health 



21 



72 53 

23 points m 
Appendix 
Guidelines 



Joint Committee, Standards for 
Evaluationai Programs, Projects, and 
Materials 



30 



about 
350-60 



155 



Evaluation Research Society, Standards 
for Evaluation Practice 



55 



55 



AERA/APA/NCME, Standards for 
Educational and Psychologicai 
Testing 



195 



195 



95 



^Refers to the number of gmdeiine statements 
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Exhibit 5. 

Standard specification element: Standard Components* 



NCE£ Eaeifx InformatioD Administratioa (ElA) 



■ Subject 

■ Effective Date 

■ Purpose 

■ Guidelines 

■ Related Standards 

■ Related Checklists, Forms, Documents 

■ Includes Checklists 



■ Subject 

■ Effective Date 

■ Purpose 

■ Applicability 

■ Required Actions 

■ References to other Standards 

■ Has Separate Checklist 

■ Has Separate Guidelines 



General AccounUng Office NCHS 

■ Each paragraph numbered ■ Outline format 

■ Major standards bolded ■ Mixed content 



Joint Committee for Edacational Evaluation Evaluation Research Society 

■ Standard ■ Two- to threc-line statement 

■ Overview 

■ Guidelines 
• Pitfalls 

■ Caveats 

■ Illustrative Case 
<i Analysis of Case 



AERA/APA/NCME 

■ Introduction to section 

■ Each individual standard is numbered 



Natioaal AasodatioB for Elenrcntary School 
Priodpab Standards for QoaUty Elementary 
Schools 

■ Introduction 

■ Standards of Excellence 

■ Quality Indicators 



^Refers to common format or elements of each standard 
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the components of eight sets of standards. There is considerable variation in their formatting, 
ranging from the two* to three-line standard statements that make up the Evaluation Research 
Society Standards to detailed illustrative cases that are part of each standard of the Joint 
Committee Standards. As currently planned each CEDCARS standard will include ( 1) an 
identification of the subject (2) a description of standard purpose, and (3) guidelines for 
implementation and, where appropriate, related checklists and references to other related 
standards. 

Standards that Include Quality Indicators 

Some standards, especially thcic used for accreditation or performance 
evaluations, include what are caUed quality indicators as part of the standard format. Those 
listing the qualities of good standards often emphasize the importance of being able to measure or 
ascertain compliance. Inclusion of ality indicators also forces the operationalizing of the 
standards. 

Although none of the seven sets of national standards that we are examining in 
depth formaUy includes specific elements that are to be checked to give indication of compliance, 
the EIA Program Offices are evaluated as to compliance and the Standards are written in such a 
way that makes compliance evaluation possible . The NCES Standards do include checklists, but 
these are in a style of offering guides that practitioners can use in planning their efforts. The Joint 
Committee Standards include a matrix in an appendix which lists the topic of each standard and 
space in which users can check whether the standard was deemed applicable and was taken into 
account, whether the standard was applicable but infeasible, whether the standard was 
inapplicable, or whether exception was taken to the standard. 

As currently proposed CEDCARS standards will stop short of specific quality 
indicators that could be used to measure compliance. Specific quality indicators may be developed 
as an additional phase to the project. 

The Technical S[>ecin€lty of the Standards 

Technical specificity refers to the extent to which the standard identifies and/or 
endorses the use of a specific research technique or practice. The extent to which standards are 
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specific is related to the breadth of the intended scope, the authority of the standards, and the 
extent lo which the topics of the standards warrant reference to specific techniques. Those listing 
the qualities of good standards often call for sundards which are both specific and general. They 
nust be general enough to have wide applicability and specific enough to give useful guidance to 
particular efforts (Brown, 1987). Observers of standards have noted that the more precise the 
standards, the more likely they will have to -cviscd. The more general the standards, the 
greater their ability to provide guidance without promoting uniformity, but the more slippery they 
became to implement and evaluate (Nowakowski, 1984). 

Among the sundards we reviewed, the EIA, NCES, AERA/APA/NCME, and 
NCHS standards generally are the most specific, and they also have the most finite or precisely 
defined scopes of applicability. Exhibit 6 gives an example of three levels of specificity for a 
standard on statistical comparison. One sees that each of the standards concerns quantitative 
comp'> isons. The Joint Committee Standard calls for appropriate and systematic analysis to 
support interpretations. The ERS Standard calls for giving indications of statistical and practical 
significance when comparisons are made, but does not specify these levels. The more specific 
NCES Standard specifies the confidence levels for hypothesis rejection, specifies the confidence 
interval levels to be included in reporting, and cautions about the use of multiple tests of 
significance. 

Determining the level of specificity can be among the most difficult problems for 
standard setters. The Evaluation Research Society Task Force on Standards discusses the 
"struggle to produce specific but nonrestrictive standards-guidance that was not also a 
straightjacket." The CEDCARS standards are aiming for a mid-level of technical specificity. 



Categories/Ranking of Standards 

It is not uncommon for standard drafters to assign levels of importance to the 
standards. Including categories of importance or indicating whether a sundard is considered 
"essential* or merely •desirable* or •important* allows for the inclusion of sundards that might be 
deemed too high or unthinkable, if all sundards had to be met. This ranking of sundards 
acknowledges the fact that some behaviors are more significant than others in promoting quality, 
and appears to be more frequently used in areas in which the sundards are used for accreditation, 
such as hospitals or universities. 
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ExhiLnt 6. 

Standard specification element: Technical SpecUicity 



■ Related to scope 

■ Related to authority 

■ Related to topics 



Examples of three levels of specificity for standard oo quantitative comparison 
Joint Committee 

■ Quantitative information in an evaluation should be appropriately and systematically analyzed 
to ensure supportable interpretations 

Evaluation Research Society 

■ When quantitative comparisons are made, indications should be provided of both statistical 
and practical signiTicance 

NCES (part oflonger standard) 

■ Confidence levels for any results of statistical tests reported in a document should it least 
90 percent before the null hypothesis is rejected. 

■ Confidence intervals around key statistics (as defmed in the analysis plan) reported in a 
document or a table should be 95 percent confidence intervals and should be clearly identLHed 
as such. 

■ There should not be more than 20 "simple" comparisons made within a bulletin or a report. 
"Simple" is defined as a t-test, chi-square test, or any other test that examines a simple 
hypothesis like the difference of means or proportions. Consideration must be given to use of 
multivariate techniques in analyses involving multiple variables, factors, or levels, and/or an 
analysis of overall error rates should be conducted where multiple comparisons and univariate 
variables are used. 



* Refers to the extent to which the standard identifies and/or explains the use of specific research techniques 
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Among the seven sets of standards we examined, only the AERA/APA/NCME 
Standards for Educational and Psychological Tests formaUy identify categories for the standards. 
NCES and EIA informally give some difference of weight to the standards by using must in some 
cases and should in others. CEDCARS standard specifications do not call for the formal assigning 
of differing levels of importance to the standards. 

Language of the Standards 

Standards are by deflnition admonitory statements, and language used in stating 
them reflects this characteristic. There is, however, some variation in the language chosen, largely 
related to the authority and role intended by the standards. Some standn-ds, while not formaUy 
designating authority, implicitly do this by the language used; i.e., incorp ^iing the words "must,** 
•^should," or "may.* In these approaches, the subject and level of authority of the standard usuaUy 
implicitly determines the verb used. Depending on the intended use of the standards, care needs 
to be taken in the use of admonitory language. Exhibit 7 summarizes the admonitory language 
used in the seven sets of standards. CEDCARS standards generally use the verb ''should." 

4. The Topics of the Standards 

This section presents an overview of the topics covered by each of the seven sets of 
standards. The standards all deal with information gathering, processing and reporting; however, 
with the exception of the ERS and the Joint Committee Standards, they differ considerably in 
intended scope. Appendix Tables A*l to A-7 list the topics of each of the standards. 

Extent of Overlap of Topics; Comparisons of the ERS and Joint Committee 
Standards 

Because they have a similar scope, several comparisons of the top .cs of the Joint 
Committee and the Evaluation Research Society Standards have been made. Daniel Stufflebeam 
(1982) compared the two sets of standards in an article published in the volume in which the ERS 
Standards were originally published. In the same volume, David Cordray ( 1982) compared the 
Joint Committee Standards, ERS Standards, and a GAO Assessing Social Program Impact 
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Exhibit 7. 

Standard specification element: Choice of Language* 



1. NCES, Standards and Policies 



Standards: must, should* is/are, can, 
need 

checklists: is/are/were 



S. Joint Committee, Standards for 

Evaluation of Educational Programs, 
Projects, and Materials 



2. EIA Energy Information 
Administration, 

Standards Manual Standards "must" or 
'are to be"; less frequently "should" 
a command structure checklist: 
is/are/were 



Standards: should 

comments: is/are /be/ will, should, must, 
do 



guidelines: command verbs 



6. ERS, Standards for Evaluation Practice 



3. GAO, Government Auditing Standards 



introductory comments: should, 

will/is/are 

Standards: should 



Standards: should (a few times: 
is/are) admonitory statements: may, 
should, need, is/are 



7. AERA/APA/NCME. Standards for 
Educational and Psychological Testing 



4. NCHS Draft Guidelines on Statistics 
and Information on Effects of the 



Standards: should (once: may, are) 
comments: is/are, should, can, do, 
may, might 



Environment on Health 

Standards: should, would, are/is, can, 
must 

introductory comments: is/are, can, 
should 

Examples of language choices 
NCES - uscof "must" 

"A prelimbary analysis plan must be developed that identifies analysis issues, major variables 
and proposed statistical techniques." 

Joint Committee - Guidelines - use of command form 

"C. Get experienced evaluators to review the timetable, and then revise it accordingly/ 

AERA/APA/NCME - use of 'should* 

"Standard 5.1 A technical manual should be made available to prospective test users at the 
time a test is published or released for operational use." 

EIA -use of "must* 

"Every primary data collection system which is operations mr*n be documented by a data 
User's ManuaL an Operations Manual and a Program Maintenance Manual." 



^Refers to whether standards are stated as recommendations, descriptions or requirements 
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Evaluations: A Checklist Approach. Both authors conclude that there is considerable overlap in the 
topics covered by the sets ofstandards.Comp^Ltuig the Joint Committee Standards with the ERS 
Standards, Stufflebeam notes that "the language of the two sets of standards is quite consistent" 
and that there is considerable, though not complete, overlap of topics. He concluded* Tor every 
ERS Society Standard there was one or more Joint Committee Sundards to provide essentially 
the equivalent." He identified four Joint Committee Standards that were not covered in the ERS 
Standards: those dealing with valuational interpretation, human interactions, balanced reporting 
and context analysis. Exhibit 8, a matrix comparing the sets of standards, is taken from his text on 
the subject. 

In comparing the content of Joint Committee and GAO 1978 Impact Evaluation 
Standards with those issued by the Evaluation Research Society (ERS), Cordray also concluded 
that there was considerable overlap and that ''there appears to be consensus (across standard setters) 
as to how evaluations should be devised, executed, and disclosed^ ( 1982, p. 70). He also notes that the 
standards are diverse in the sense that they do not espouse one individual evaluation strategy over 
another (for example, case studies vs. randomized experiments), but instead demonstrate that the 
'^members of the evaluation industry appear to have a model articulating what constitutes sound 
practice that crosses disciplinary boundaries and types of evaluations" (p. 73). Exhibit 9 is a copy 
of a table enumerating the issues addressed in the ERS Standards and the corresponding 
standards from the other sets. 

Topk Overlap: Other Comparisons 

The other sets of standards are different enoufji in scope, level of specificity, and 
organization to make an analysis of topical overlap less useful We present some comparisons 
here, only as a help in cross*referencing the standards. It should be noted that the decision about 
whether there is a topical overlap is a subjective one. Other attempts to judge overlap have shown 
inconsistency among the differing judges' decisions (Stufflebeam, 1982). 

Exhibit 10 examines the extent to which NCES Standards cover the topics included 
in the Joint Committee Evaluation Standards. While we judged that most of the topics of the Joint 
Committee Educational Evaluation Standards are covered by NCES in some fashion, they are 
often covered as subtopics within a larger standard, and there is a very different emphasis. In 
other words, the topics may be similar, but the focus is quite different. The topics covered by the 
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Exhibit 8. A Comparison of the ERS and Joint Committee Standards (taiccn from Daniel Stufflcbeam. A Next Step. D 
to Consider Unifying the ERS and Joint Committee Standards," 1982, table 1) 
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1 The numben in ihe cells of the matnx refer to the 55 ERS standards. 

2 The stars (') in (he cells of the mairu denote »iiich of the 30 Joint Commmee standards are most relevant to each section of ERS standards 
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Exhibit 9. Comparison of the Content of the Standards Issued by the Evaluation Research Society (ERS), Jomt Commitiee 
Standards for Educational Evaluation, and General Accounting Office (GAO) (from David S. Cordoray, "An 

A«eMinent of the Uttlitvnf the ERS Stand ards.' 19S2. tahl.> U ^ 

ERS (1982) Joint Ccmmiaee (1981) QAO * (1978) 



FORMin JVTIQN AND NEGOTIATION 



1. Purposes and characteristics of the program 


Dl 


A2.1-A2J 


2. Audience, needs and expectations 


Al 


Al.l-AlJ 


3. Type, objectives, range of activities for evaluation 


A3 A4 


A1.4 A3 All All 


4. Sound, prudent and ethically responsible cost estimate 


C8 


• 


5. Cost -benefit of evaluative information 


B3 


• 


6. Feasibility of the evaluation 


81 82 


A2.1-A2.4 A7.1-A7.4 A8 


7. Restrictions on data access or dissemination 


C4 


E2.2 


8. Conflict of mterest 


C2 


• 


9. Rights and welfare of parties 


C5 


E2.2 


10. Technical and financial accountability 


C8 




11. Formal agreements 


CI 


A7.4 A8.1 A8.2 


12. Capabilities 


A2 


81.2 81J 81i 


STRUCTURE A>JD DESIGN 






13. Approach to evaluation 


A4 


A3.2 A3.4 D\4 


14. Estimating c (Tects 


D8 


A3.4 C1.1-C1.3 C3 A4.1 


15. Sampling methods 


D4 


A4.2 A4.3 A6 C2 


16. Reuabtuty and vahdity of measures 


D5 E)6 


A5 A6 


17. Appropriateness of procedures and instruments 


D3 D5 D6 


A3.1 A3.2 A6 


18. Cooperation 


CI 


A8.1 A8.2 


DATA COLLECTION AND PREPARATION 






19. Data collection and preparation plan 


• 


• 


20. Departures from original plan 


D7 


El.l 


21. Staff competency 


D7 A2 


BU B13 B13 


22. Preservation of human dignity 


C6 


• 


23. Verification of reliability and validity 


D4 D5 D6 


A5 


24. Sources of errors 


D7 


B1.6 B1.7 B2.1 


25. Biased data coUectioa 


D7 Dll 


A5 


26. Minimum disruption 


Bl 


• 


27. Risks and informed consent 


C5 


• 


28. Unauthorized release 


C5 


E22 E2.1 


29. Complete documentation 


EM 


E12 


30. IrreoTverable Ires of data 


• 


• 



'Indicates not covsred 

^Tbese GAO standards, published in 1978, are not those indudcu in this review. 
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Exhibit 10. A comparison of topics of standards issued by the ioint Committee* Evaluation Research Society 
(ERS) and the National Center for Education Statistics (NCES) 



Joint Committee (1981) ERS (1982) NCES (1987) 

A. UUIity 

1. Audience Identification 2 01*01 

2. Evaluator Credibility 6, 12 02-01 02-02 

3. Information Scope and Selection 2, 13 01*01 

4. Valuational Interpretation 14 03-04 04-04 

5. Report Clarity 39,40,41,42,43,44,45 04-02 

6. Report Dissemination 46, 47 04-05 

7. Report Timeliness 50 04-05 

8. Evaluation Impact S, 46 04-05 04^7 

B. Feasibility Standards 

1. Practical Procedures 6 01-01 (Checklist) 

2. Political Viability 6 • 

3. Cost Effectiveness 4/5 • 

C. Propriety SUndards 

1. Formal Obligation 7, 10, II 02*01 



7 



Conflict of Interest S 



3. Full and Frank Disclosure 7,43,44,45 04-01 

4. PubUc's Right to Know 7, 46, 47 04-05 04-06 

5. Rights of Human Subjects 9 03*02 04^3 

6. Human Interactions 22 04-03 

7. Balanced Reporting 41, 53, 54, 55 04-02 04-04 

8. Fiscal Responsibility 10 01*01 (Checklist) 

D. Accuracy Standards 

1. Object Identification 3 01*01 

2. Context Analysis 6 01*01 

3. Described Puiposc and Procedures 1 01*01 04-01 

4. Defensible Information Sources 43,45 01*02 03*02 05*02 

5. Valid Measurement 23 01*02 03-02 

6. Reliable Measurement 23 01*02 03-02 

7. Systema^Jc DaU Conlrtrf 24 03-02 03-03 03-04 

8. Analysis of Quantitative Information 31, 32, 33, 35, 36, 36, 37 03-04 04-04 05*01 

9. Analysis of Qualitative Information 31-38 * 

10. Justified Conclusions 39 03-04 

11. Objective Reporting 39 04-04 05-02 



* Indicates not covered 
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Joint Committee that are not mentioned by NCES include political viability^ cost effectiveness, 
conflict of interest, and analysis of qualitative information. 

Exhibit 11 presents a comparison of the NCES topics and subtopics with those 
covered by EIA, the Joint Committee and ERS. Considerable overlap occurs between NCES and 
EIA standards; however, the EIA standards are more focused on developing and maintaining data 
systems, while NCES standards are more concerned with one-time or periodically recurrent 
surveys. We judged that the NCES Standards cover a number of topics that are not covered at all 
by the Joint Committee or the ERS Standards, except on a very general level. The topics not 
covered by the Joint Committee primarily concern detailed standards related to survey research 
and statistical methods covered in detail by both NCES and EIA. 



Standards that Call for Documentation 

In an overview of aU the standards, the large percentage of standards that deal with 
providing adequate description and documentation of procedures is immediately apparent. For 
example, of the 55 Evaluation Research Society Standards. IS have a standard on providing 
documentation or descriptions. This formalization of procedures is seen as serving two purposes: 
first, the researcher is forced to clarify and plan data coUection and reporting activities in detail, 
and second, the person using the results is provided with the necessary information to assess the 
quality of the data. 
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Exhibit 11. Coverage by ERS and Jobt Committee Standards of topics included In NCES Standards and Policies 





EtA 


ERS 


Joint Committee 


01. SUndards for PUoniof 








01<Oi Suodard for plaDoing 








Adoption of 0MB standards 

Survey must be justified 

Related surveys reviewed/avoid duplication 

Consultation with users 

Must have study plan 

Must have analysis plan 

Must have publication plan 

Checklist 


02-03 
CK IV, 0M)1 
C4-01 

02-02, 05-01 
05-01 
05-01 
CK V 


5 

18 
D 
35 
48 


B3 

A1,B1,B2.C6.D1 
A2,B1.C1.C2,C5.D3 
C1,D8.D9 
A6A7.Dn 


01-02 Standard for independent evaluation of surveys 








Independent evaluation incorporated mto plan 
Include experimenti to identify sources of nonsampling error 
Items identified that can be used m mtemal checking of data 
Sources outside data should be identified to check validity 
On-going research plan built mto study design 


CKVI 
CKVI 
CKVI 


25 
23 


A2^6.D3.D9.D1I 
A4.D10 


02. Standards oo RFP Development/Contract Monitoring 








02-01 Standard on RFP dcvclopmeot for surveys 








RFP specify technically sound design with adequate information 
Project Officer responsibilities 
Objective evaluation criteria 


• 


39 


CI 


02-02 Standard for mooitoring survey contracts 








Administrative responsibilities specified 
Quality assurance responsibilities specified 


02-01 




CI 


02-03 Standard for malntaliiiaf contract files 








Contenu and responsibility for files specified 








03. Sundards for ImplenitBUtioo/Data 
CoUectioa/ProceMing 


04 






03-01 SUndard for tcstlDf data collcctioa instniments 
and data coUcctloa systenu 








Review of questionnaires 

Internal pretest for feasibility of questionnaires 

Pretest in the field 

Provision for successive pretests 

Tpctinff mi»i)iiv1c of A%t% cnllection 

Testmg for absuaction of records 
Quality control procedures tested 
Checklist 


CK IV, CK V 

CKIV 

CK IV 

CKVI 
CK IV 




A3 
D5 
D5 

D5,D6.D7 

D7 
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Exhibit 11. Coverage by ERS aod Joint Committee Standards of topics included in the NCES Standards and Policies 
(continued) 





EU 


ERS 


Joint CoEQiuittce 


03-02 Stiudard for cdoaitlooal tests 








Adoption of APA standards 








Evidence of validity presented 






D5 


Evidence of reliability presented 






D6 


Procedure for administration specified 








Special testing for special populations 








Rights of test takers proteacd 




28 




Uoecxust 








03*03 Standard for miaimlainf survey 








nonrespoBsc 








Defmition of nonresponse 








Efforts required to collect information not initially obtained 


04-02 






iicra nonresponse enons 








Design considerations to minimize nonresponse 








Relationship of instrument design to nonresponse and documentation 








of efforts to minimize nonresponse 








03«O4 Benchmarks for data collectloot processing, 








and analysis 


4(01-06) 






Exceptions to benchmarks 








Target overall response rates specified 


04-02 






Stratum response rates specified 


04^32 






Critical variable response rates specified 








Deviations from target response rates anticipated 


04-02 






Analysis of impact of low rates 


04-02 






Use of variables with high rates 








Weighting the sample for nonresponse procedures 


04-02 






Cautions on weighting for nonresponse 


04-02 






Minimum necessary cell size for reporting data in tables 








Confidence levels for use in statistical tests specified 


05-04 






Confidence interval level for use around key statistics specified 


05-04 






use oi muiupic compuison uniics spccmcy 








Method for calculating response rate specified 


CKVI 






Item response rate calculation procedures specified 


CKVI 






i^ovcragc rate caicuiauon proceourcs spcnucq 








03-05 SUndardforeodcs,abbrtviatlo«s 








and acrooyw 


02-02 






Adoption of standard Federal Informatioa Processing Standards (HPS) 


02-01 






Directions on coUectioo of race and ethnicity 


02-03 






Designations not acceptable 








Need for race and Hispanic origin 








Approved list of acronyms adopted 


02-02 






Standard glossaries of definitions adopted 


02-01 






Style manual adopted 


02-01 
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Exhibit 11. Coverage by ERS and Joint Committee Standards of topics included in the NCES Standards and Policies 
(coDtmued) 





£U 


ERS 


Joint Committee 


03«06 SUndard for rounding 








Decision for appropriate number of significant digits 
Using data with diftenng signincant digits 
Rounding with regard to standard erro.s 
Rules for rounding 

Need for explanation of differences due to rounding 


05-07 
05-07 
05-07 
05-07 
05-07 






03-07 Sundard for daU Upe preparation 






D7 


Data set name requirements 
Types of acceptable tape formats 
File structure consistent with checklist 
Form that must be completed 

Machine readable Tile and file description and record layout must be 
on all tapes 

Documentation must be included 


05-09, 04-06 
05-09,04-06 
05-09, 04-06 
05-09,04-06 

(\C /V^ (\A f\£. 

05-09, 04-06 
05-09, 04-03 






04. Reiease/Pubiication of DaU 


05 






04-01 Standard for survey documentation 


05-09 






Documentation must allow non-statistical user to understand 

limitations and quality of data 
Documentation should include abstract, status of data^ 

methodology, sampling and non*sampling errors, 

and guidelines for urging the data 
Checklist 




40, 41 
43,44,45 


AS. D8 
D3,D4 


04-02 Standard for technical documentation in 
data releases 


03-03 






Level of documentation and technical nature of discussion 
will vary with audiences 
Specification of contents of major reports 
Key statistics should be followed by confidence mtervals 
Specification of contents of short reports 
Press release includes sampling error 
Public release tapes confirm to requirements 
Requirements for EJ>. TABS 
Proper doaunentation of data used from other sources 




DIO 


A5,D8 
A6 

A6 


04-03 Standard for maintainini confidentiality 


05-06 






Respondents must be told whether participation is voluntary 
Respondents must be informed of confidentiality provisions 
RFP's must describe extent to which confidentiality must be 
maintained 

Where absolute confidentiality is promised, respondents data 

must not be accepted with identifiers 
Care in reporting data in cells to make sure respondent cannot 

identify; use of suppression in cells with one or two respondents 


05-06 




C5 

C4,C5 
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Exhibit 11. Coverage by ERS and Joint Committee Standards of topics included in the NCES Standards and Policies 
(continued) 





EU 


ERS 


Joint Copjmittee 


04^ SUndard for tabular prescntaUoa 


CKvn 






Title reqmrements 

L^lTCUH^ 1 CijUUCinCUia 

Identification of sources 
Kow ouu column loiajis 
Bases for percent 

Standard errors or confidence intervals 
Suggestions for use of graphs 


CK VII 
CK 05-01 
CK VII 

CK VII, 05-04 
CK Vn, 05-05 






04«O5 Standard fur dlsscmioatkMi of survey data and results 








Planiung for dissemtnation i one concurrently with survey planning 
Fmdings published on fixed .^edule 
Keep publication schedule 

onortcst possiDie interval, clean oata tape in o tnontns 
Preparation of articles for journals 
Publications 100 percent accurate* 


05-03 






04-06 Standard for timely processing and release 
of data and dau tapes 








Published schedule for recunent surveys 

L/incr oeaQiines niusi oe mci lo meci puoucauon oeauiines 

All reporting units notified of publication deadlines ahead of time 

Shortest interval exists between data collection and tape preparation. 

Clean tape in 6 months after the end of data collection 

Schedule with information services 

Data not released without ofiidal approvals 


05-02 






04*07 SUndard for release of staUstical dau 








Data not released in any form without approval 
Ke lease scneauies avauaoie lo puouc 
Goal: no more than two revisions of data set 

C Intv fl^ifa ^r/^m NI^f*Q rf^l^ftCM^ /Ittts tan^c mav ttci»i4 in 

wuiy U4iui iruin oiii*aai i^^co r cmicu umim ispco tndj uc uscu ui 

publications 

Pplpac^rl flip IrPfV flf r^Amniif^r rVnlM 

Woricing files archived for at least 18 months 


05-02 
05-08 






05. SampUof aad Noa^SampUng Ennor 








05«01 c^uadard for treatiMil of noorespoose 








Best efforts to minimize nonrespoose 

If more than 30 percent of key items ire missing, consideration to delete 
record and make weighting adjustment for nonresponse 
Imputation should be clearly indicated on the data tape 
For continuing surveys for categorical data, use data on the last surve> 
to impute data 


04-02 
04-02 
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Exhibit 11. Coverage by ERS and Joint Committee Standards of topics mcluded lo the NCES Standards and Policies 
(continued) 





EU 


ERS 


Joint Committee 


For continuous data for contbuing surveys, if no trends in data, use 
last report 

For continuous data, if trend, impute using regression 
Use of ratio adjiistments 
Use of hoc deck adjustments 
Do not zero flit 








05-02 Standard for estimating sampling error 


05-06 






Consideration of methods of variance estimation, use of replicates should 
be part of design 

Estimation should make use of data from other sources 
Replication techniques research to determbe the number of random 
groups to be used 
Generalized vanance procedures 
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Topic Spedncity 



One criticism of the Evaluation Research Society Standards was that the 
methodological guidelines were far too broad to be instructive. Berk ( 1982) notes that there were 
many instances where increased specificity was both desirable and possible. He states that '^it is a 
simple matter to formulate technical guidelines that seem to pose no threat to flexibility and 
innovation and that still seem to capture important aspects of sound methodological practice." He 
gives the following examples of guidelines more specific than those put forth by ERS that he would 
recommend (1982, p. 64): 

1. Response rates should be calculated and reported for all evaluations resting on 
survey data. 

2. The potential role of sample selection bias should be analyzed and reported. 

3. Analyses of attrition should be routinely undertaken and reported. 

4. The impaa of outliers should be routinely analyzed and reported. 

5. The assumptions underlying all statistical procedures should be reviewed and 
reported. 

6. The likelihood that these assumptions are violated should be reported along with 
the possible consequences of these violaclons. 

7. Results based on a priori theory and model specifications should be clearly 
distinguished from results resting on post hoc models and exploratory analyses. 

8. Results resting on multiple significance tests should be properly discounted, or at 
least the fact that multiple tests have been undertaken should be reported. 

9. Procedures used for missing data should be reported and critically reviewed. 

10. Computer programs used at any stage of the research should be described along 
with any suspected problems (for example, vulnerability to rounding errors.) 



5* Sumnuury of CEDCARS and Prtsentation of a Total DaU Quality Improvement Model 

In this section we summarize the requirements/characteristics of the CEDCARS 
project and present a formal conceptual analysis that underlies the development of the principles 
and topics included in the CEDCARS standards. This conceptual framework borrows heavily 
from the work of W. Edwards Deming and also from the work of the Joint Committee. We are 
calling it the Total Data Quality Improvement Model. 
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What Is the Scope of Data Coverage and the Definition of a SUndard for 

CEDCARS? The CEDCARS Standards arc not limited in scope to survey data, but rather cover 
aU types of educational data, e.g.. records and documents data, data for evaluation, data for policy 
studies, data for accountability studies, data for educational achievement through assessment and 
testing, data for the improvement of education, and data for all types of education indicators. For 
the CEDCARS project, a Standard is defined as '*A principle for guiding the conduct and assessing 
the quality of a data-related inquiry to which those who design, provide, collect, process, analy-i^e, 
report, and u^ Jucational data generally agree." 



What Are the Requirements of CEDCARS? 



The requirements for the Standards for education data are as follows: 

■ They must be cooperatively developed through an iterating process to reach 
consensus. 

The importance of cooperatively developing Standards for education data relates to 
the political and organizational struaure of the educational system in the U.S. a 
State and locally controUed and fmanced system. Thus, it is very important that 
concerned individuals in schools, school districts. State education agencies, and 
private school associations and schools be a part of the process of defining the 
principles and guidelines that can be used to improve the quality of the data they- 
provide. 

■ They must reflect the consensus of best practice and be technically adequate. 

■ They must be comprehensive in the coverage of the contents of education data and 
comprehensible to all of the many potential users so that they will have wide 
a pplicability to the uses. 

■ They must not inhibit innovation and development as it relates to education data 
and must be capable of being changed over time. That is, there is continuing 
development that does not end, creating constant effort to improve the quality of 
the Standards* 

In the development of the CEDCARS Standards, we have been concerned with the 
relationship of the Standards to definitions of data elements. It has been an explicit requirement, 
as directed by the task force of the project, that we focus on defining the characteristics of good 
data elements, but not mandate specific definitions of data elements that the local and State 
education agencies must use. 



ERIC 



41 

44 



Who Are the Intended Users oT CEDCARS? 



The Standards for CEDCARS have many intended users, including personnel of 
State education agencies, locd education agencies, Federal agencies, private and pubix schools, 
research organizations, professional organizations, academic educational researchers, 
policymakers, and school managers. The CEDCARS project suff estimates that a lOial of about 
20.000 individuals in State and local education agencies can improve quality of thev data or 
otherwise benefit by being primary users of the Standards. 

The cooperative process of developing the standards, or "ownership by 
involvement" as we have called it. is essential to the standard development process. The 
investment of those who are potential users of the Standards is necessary because there must be 
agreement on the Standards before data quality improvement can occur. The importance of 
people using the Standards lies in the acceptance of the extended data process by the members of 
the 87,000 schools. 15.000 school districts, and 57 State and territorial agencies in the U.S. 

What Are the Intended Uses of CEDCARS? 

The CEDCARS Standards will be used for many purposes. Some of the primary 
intended uses are as follows: 

■ As tools for planning data coUection and reporting; 

■ As tools for evaluating data coUection and reporting; 

■ As a means of justifying implementation of improvements in data couection and 
reporting; 

As tools for training new practioner and providing technical assistance to existing 
practioners; and 

■ As a way to professionalize the endeavors of education data coUection activities 
which wiU define BEST practices . 

Total Data Quality Inprovement Model 

The topics and content of the proposed CEDCAR Standards were developed in 
conjunction with consideration of the phases of data quaUty consciousness. These phases, which 
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for any system are inspection, prevention, and Improvenient, are also of fundamental importance 
to the development, dissemination, and implementation of standards for the improvement of data 
quality. 

Let us consider two hypothetical cases concerning organizations that lack data 
quality consciousness. One agency ignores data quality standards altogether, and collects, 
processes, analyzes, and reports data of poor quality. A second agency also ignores standards in 
collection of data but invests considerable cost into the inspection of data before they are released 
through cleaning operations and other intensive and expensive efforts. In both cases the initial 
collection of the data will be of poor quality because the processes that produced the faulty data 
have not been curtailed, and therefore no safeguards exist to prevent inaccuracies or errors in the 
data. The data of Case II will r^ault in data of higher quality, but will be more expensive to collect 
and process due to the wasted resourses in the inspection of the data for faults or errors due to the 
lack of prevention in the initial design and implementation phases of data collection. The 
consciousness of the culture or system for collecting and reporting data of these two agencies is in 
need of changes for the improvement of data quality. It is for this reason that we must consider 
the components or subprocesses of the "extended data process." 



What Is the Extended Data Process? 

The extended data process (see Exhibit 12) specifies that the initial task is to 
identify the need for or the purposes for which the data are to be coUected and later used in an 
analytical and reporting plan. Our conceptualization of the next component of the extended 
process is the design process, which includes sample design if sampling is applicable, designs for 
the measurements (questionnaires) of the survey, data collection, processing, analysis, and its 
report. The additional components of coUection, processing, analysis, and reporting are 
considered to be the operational subprocesses of the extended data process. These six 
functional/temporal concepts were used as topic groups in the development and writing of the 
CEDCARS Standards. Exhibit 13 presents the list of proposed CEDCARS topics reflecting the 
extended data process. 
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Exhibit 12. Components of the Extended Data Process 



F 
E 

E L 
D O 
B O 
A p 
C 
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NEED^ 

c 

DESIGN ^ 

i : ' _^ 

COLLECT PROCESS ANALYZE REPORT^) 



I 

USE 
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Exhibit 13. Cooperative Education Data Collection and Reporting Standards 



LIST OF TOPICS 



I. Identification and Coordination of Data NEEDS 

1. Identiiy primary target audience(s) 

2. Justify each data collection program 

3. Conduct appropriate review and approval processes for data collection 

4. Coordinate data collection and analysis programs 

5. Standardize definitions, data collection schedules and procedures, and data processing 

6. Review and monitor ongoing data collection periodically 

II. Study DESIGN 

1. ^jentification of the Study Questions 

2. Selection of Appropriate Methodologies to Collect Data 

3. Define Data Elements 

4. Are the Data Worth Collecting 

5. Selection of Appropriate Measurement Scales and/or Operationalization 

6. Designing the Data Collection Instrument 

7. Field Test/Pretest of Survey Instruments, Forms, and Procedures 

8. Written Stucty Design 

III. Data COLLECTION 

1. Preparation/Plan for Data Collection 

2. Documentation 

3. Staffing and Training 

4. Ethical Treatment of Respondents 

5. Minimize Burden 

6. Efforts to Maximize Response 

7. Quality Control 

8. Implementation of Data Collection Procedures 

9. Guidelines for Completing Survey or Study Forms 

IV. Data PROCESSING 

1. Scope of Project 

2. Systems Design^ Development, and Testing 

3. Data Preparation 

4. Disaster Recovery Plan 

5. Prognun and Data File Back*Up 

6. Data Storage and Retrieval 

7. Retention of Data Files and Program 

8. Security and Confidentiality 

9. Data Processing Documentation 

10. Usabili^ 

1 1 . Evaluation of the Data Processing System 
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Exhibit 13. Cooperative Education Data Collection and Reporting Standards (continued) 



V. ANALYSIS 

1. Weighting of Survey Data 

2. Nonresponse Adjustments 

3. Use of the Descriptive and Analytic Techniques 

4. Sampling Errore 

5. Nonsampling Errors 

6. Statistical Significance 

7. Validity and Reliability of Instruments 

8. Documentation of Data Collection and Statistical Techniques 

VI. REPORTING 

1. Report Organization and Writing 

2. Report Review 

3. Release of Data 

4. Dissemination 
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What Are General Qualities or Conditions of Dau Quality Impitivement and 
Their Relationship to CEDCARS and the Extended Data Process? 

We describe the general qualities of data by a data quality diamond of four conditions or •'bases*' 
accuracy, feasibility, utility, and proprietary (Exhibit 14). Under accuracy, the home base on top of 
the diamond, we include comparability and uniformity which links to the words of the 
Congressional mandate that established the CEDCARS project of the Cooperative Educational 
Statistics System Forum. Also included under accuracy are reliability and validity, which are 
concepts familiar to educational researchers. Feasibility, which includes resource available, 
burden of data collection, and schedule constraints are on the left or at "first base." Utility, which 
relates to the use of the data, is at the bottom of the diamond or "second base," and includes the 
critical element of timeliness. The proprietary element of data, whicii includes qualities that relate 
to confidentiality and disclosure, are placed on the right side of the diamond at "third base." Inside 
the diamond, we have included the data quality standards, under the components or quality 
element of the extended process, needs, design, coUection, processing, analysis, and reporting. The 
data quality diamond portrays a holistic approach to total data quality improvement. For example, 
data feasibilty is possible, but it depends on decisions about accuracy required, the ultimate uses of 
the data, resources available, and any constraints that might relate to confidentiality or disclosure 
in the coUection and release of the data. According to the phases of data quality consciousness, we 
know that ALL data have mistakes or errors, but we continuaUy work on the components of the 
extended process to PREVENT then IMPROVE it by use of written standards to document a 
BEST practice. 

What Is Data Quality Improvement? 

By definition, data quality improvement is the never-ending task of decreasing the 
difference between best practice (standards) and current practi;c. To tipply this definition of data 
quality improvement one must first assess the current practice by identifying the need or purpose 
of the data and how the design was developed, and by assessing the operational aspects of 
collecting, processing, and analyzing data and how they are reported or published. We must 
compare the results of the "current practice assessment" to the principles, guidelines, and checklists 
in the standards which represents BEST practices and suggest or recommend changes to the 
extended process that lead to quality improvements. 
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Exhibit 14. Qualities of Data 
ACCURACY 

Comparability, Uniformity, 
Validity, and Reliability 
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.PROPRIETY 

Confidentiality 



Disclosure 



ERIC 



♦ 

UTILITY 

Timeliness 
51 



Exhibit 15 provides three cases of the data quality improvement. Case I 
demonstrates poor quality arising because the current data practice docs not conform with BEST 
practice. Case II shows the improved data quality that results when some of the current practices 
overlap with the BEST practices. Case III presents the ideal situation in which current praaices 
are the BEST praaices. Ultimately, the major objective of the CEDCARS at the implementation 
phase is to use the project standards to assess the difference between current practice within a 
data program to the best practices articulated by the standards. 



How Should CEDCARS Be Used and Implemented? 

Among the uses of the CEDCAR Standards will be to provide training and technical assistance to 
education agencies so they can design and conduct data quality studies or asssessment to improve 
their current practice relative to the best practice as identified or documented in the written 
CEDCAR Standards for education data. As a part of this assessment, consideration will be given 
to the types of data quality studies, such as data quality studies for design/redesign, often referred 
to as methodological studies, conibrmance and performance studies. Conformance data quality 
studies occur during the operational components of coUection, processing, and analysis; e.g., data 
editing. Performance data quality studies are used after the data coUection has fmished and the 
analysis and report has been completed. Performance data quality studies are used to assess 
whether the data meet th" user needs for policy making and/or management decisions. 

CEDCARS draft Standards are scheduled for completion in September 1991. A 
draft has been completed and is available for review upon request to David Bayless, Chair of the 
CEDCARS Task Force. 
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Exhibit 15. Three Cases of Data Quality Improvement 
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APPENDIX A EXHIBITS 



Exhibit A* I. Topics included in the Joint Committee Standards for Evaluation of Educational Froffwns» Projects, and Matenals 



A. UtUltjr 

1. Audience identification 

2. Evaluator credibility 

3. Information scope and selection 

4. Valuational interpretation 

5. Report clarity 

6. Report dissemination 

7. Report timelines 

8. Evaluation impact 

B. Feasibility Standards 

1. Practical procedures 

2. Political viability 

3. Cost effectiveness 

C. Propriety SUndards 

1. Formal obligation 

' 2. Conflict of interest 

3. Full and frank disclosure 

4. Public's right to know 

5. Rights of human subjects 

6. Human interactions 

7. Balanced reporting 

8. Fiscal responsibility 

D. Accuracy Standards 

1. Object identification 

2. Context analysis 

3. Described purposes and procedures 

4. Defensible information sources 

5. Valid measurement 

6. Reliable measurement 

7. Systematic data control 

8. Analysis of quantitative information 

9. Analysis of qualitative infonnatiott 

10. Justified conchttions 

1 1. Objective reporting 



Exhibit A*2. Topics included in NCES Standards and Policies 



01. Suadards for Ptoulog 

0141 SUodard for pluttlsf 

Adopdoa of OMB ^^uidards 

Survey must be justified 

Related surveys revieived/avotd duplication 

Coosultalioii with users 

Must have study plan 

Must have analysis plan 

Must have pubUcatioo plan 

Checklist 

01- 02 Standard for independent evaluation of surveys 

Independent evaluation incorporated into plan 
Include experiments to identify sources of nonsampUng error 
Items identified that can be used in mtemal checking of data 
Sources outside data should be identified to check validity 
On-going research plan built into study design 

02. SUndards on RFP Deveiopnent/Contract Monitoring 

02- 01 Standard on RFP development for surveys 

RFP specify technically sound design with adequate information 
Project Officer responsibilities 
Objeaive evaluation criteria 

02-02 Standard for monitoring survey contracts 

Administrative responsibilities specified 
Quality assurance responsibilities specified 

02-03 Standard for aMtntai ' ^ ::ontract flies 

Contents and responsibility for files specified 

03. SUndards for ImpieMenlaUon/Dnta Collection/Processing 

034)1 Standard for testing data coiicctioa InstrumenU and data collection systems 

Review of questionnaires 

Internal pretest for feasibility of questionnaires 

Pretest in the field 

Provision for successive pretrsts 

Testing method4( of data coU'xtion 

Testing for abstraction of records 

Quality control procedures tested 

Checklist 
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Exhibit A-1 Topics included in the NCES Standards and Poikies (cootinued) 



03^ Smdard for cdvailiouJ tcsU 

Adopcioo of APA stindards 
Evidence of viUdity pretented 
Evidence of reliability presented 
Procedure for administntion spedfied 
Special testing for special populations 
Righu J test takers protected 
Checklist 

034)3 Standard for minimizlnf survey nonmponse 

Defmition of oonresponse 

Efforts required to collect information not initially obtained 

Item nonrespoQse efforts 

Design considerations to minimize nonresponse 

Relationship of instrument design to nonresponse and documentation of efforts to minimize nonrespoose 

03«O4 Benchmarks for data collection, processing, and analysis 

Exceptions to benchmarks 

Target overall response rates specified 

Stratum response rates speoRtd 

Critical variable response rates specified 

Deviations f;om target response rates anticipated 

Analysis of impact of low rates 

Use of variables with high rates . 

Weighting the sample for nonresponse procedures 

Cautions on weighting for nonresponse 

Minimum necessary cell size for reporting data in tables 

Confidence levels for use in statistical tests specified 

Confidence interval level for use around key statistics specified 

Use of multiple comparison limits spedfied 

Method for calculating response rate specified 

Item response rate calculation proctdiaes specified 

Coverage rate calculation procedures specified 

03-05 Standard for codett abbreviations and acronyms 

Adoption of standard Federal Information Processing Standards (FIFS) 

Directions on coUection of race and ethnicity 

Designations not acceptable 

Need for race and Hispanic origin 

Approved list of acronyms adopted 

Standard glossaries of definitions adopted 

Style manual adopted 
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Exhibit A-1 Topics included in the NCES Standards and Poiicies (continued) 



Standard for roanding 



Decision for appropriate number of significant digits 
Using data with dl^ering significant digits 
Rounding with regard to standard errors 
Rules for rounding 

Need for explanation of differences due to rounding 

03- 07 Standard for data tape preparation 

Data set name requirements 
Types of acceptable tape formats 
File structure consistent with checklist 
Form that must be completed 

Machine readable fde and fUe description and record layout must be on all tapes 
Documentation must be mduded 

04. Reieasc/Publication of Data 

04^1 Standard for survey documentatif^n 

Dociunentation must allow non-statistical user to understand limitations and quality of data 
Documentation should include abstract, status of data, methodology, sampling and non-sampling errors, and 
guidelines for using the data 
Checklist 

04- 02 Standard for technical documenUtlon in data releases 

Level of documentation and technical nature of discussion will vary with audiences 

Specification of contents of major reports 

Key statistics should be followed by confidence intervals 

Specification of contents of short reports 

Press release mdude sampling error 

Public release tapes confirm to requirements 

Requirements for E.D. TABS 

Proper documentation of data used from other sources 

04-03 Standard for aalntalnlng confidentiality 

Respondents must be told whether participation is voluntary 
RespoodeaU must be informed of confidentiality provisions 
RFPs must describe extent to nrfiich confidentiality must be maintained 
Where absolute confidentiality is promised, respondents data must not be accepted with identifiers 
Care m reporting data in ceils to make sure respondent cannot identify; use of suppression in cells with 
one 0* two respondents 
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Exhibit Topics included in the NCES Standards ana Policies (continued) 



04-4^; Standard for tabular preseoutioa 

Title requirements 
Labeling requirements 
Identification of sources 
Row and column totals 
Bases for percent 

Standard errors or confidence intervals 
Suggestions for use of graphs 

Standard for dissemination of survey data and results 

Planning for dissemination done concurrently with survey planning 
Findings published on fixed schedule 
Keep publication schedule 

Shortest possible interval, deasi data tape in 6 jionths 
Preparation of articles for journals 
Publications 100 percent accurate 

04^ Standard for timely processing and release of data and data tapes 

Published schedule for recurrent surveys 

Other dradlinf5 must be met to meet publication deadlines 

All reporting units notified of publication deadlines ahead of time 

Shortest interval exists between data collection and tape preparation. Clean tape in 6 months after the end 
of data collection 
Schedule with information services 
Data not released without offida! approvals 

04<07 Standard for release of statisticai daU 

Data not released in any form without approval 
Release schedules available to public 
Goal: no more than two revisions of dau set 

Only data from official NCES released data tapes may be used in publications 
Released file kept at Computer Center 
Working files archived for at least 18 months 

05. Sampling and Nott«SampUat Error 

0541 Standard for treatment of nonresponse 

Best efforts to minimsEt nonresponse 

If more tlian 30 percent of key items are missing» consideration to delete record and make weighting 
adjustment for nonresponse 
Imputation sbouM be dearly indicated on the dau tape 

For continuing surveys for categorical data, use data on the last survey to impute data 
For continuous data for continuing surveys, if no trends in data, use last report 
For continuous data, if trend, impute using regression 
Use of ratio adjustmetits 
Use of hot deck adjustments 
Do not zero fill 
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Exhibit A-2. Topics included in the NCES Standards and Policies (continued) 



05-02 Staadard for estimatliif samptinf error 

Consideration of methods of variance estimation, use of replicates should be part of design 
Estimation should make use of data from other sources 

Replication techniques research to determine the number of random groups to be used 
Generalized variance procedures 
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Exhibit A-3. Topki covmd *m Standanis for EvaluMon Fw 



FonnulatiM aad Ncfotifttios 

1. Pmpottcs ind duracteristks specified 

2. Cliems and users idemified 

3. Type of eviluatkMi identified 

4. Estimate of cost prepared 

5. Value to justify cost 

6. Feas3)ility of evaluatioo assessed 

7. Restricuoiis on acceu to data specified 

8. Conflicts of mterest justified 

9. Respect for all parties* rights/welfare 

10. Fuundal and technical accountability 
IL Formal agreements in writing 

12. Evaluators' professional qualifications/resources 

Structure and Design 

13. Clear design specified 

14. Problem of effects of nontreatment 

15. Sampling methodology 

16. Measurement methods reliability/validity 

17. Justification of procedures and instruments 

18. Cooperation of those involved in evaluation secured 

Data Collection and Preparation 

19. Data Collection Plan 

20. Provision for departures from origimJ design 

21. Staff trained and supervised 

22. Rights of mdividuals protected 

23. Validity and reliability verified 

24. Analysis of sources of error 

25. Safeguards for biases 

26. Minimum disruption 

27. Rw^ew of risk of adverse effects 
23. Confidentiality safeguards 

29. Documentation of each item of data 

30. Safeguards against data lou 

Data Analysis and In te rptt ta tloa 

31. Analytic procedures match purposes 

31 Describe and present analytic profxdures 

33. Relationship of analytic procedures to measures 

34. Uiuts of analysis appropriate to dau collection/use 

35. Justification for analytic procedures 

36. Documentation for anaiyps 

37. Statistical and practical ^ignificancff of comparisons 

38. Cause-and^ffect interpreution 

39. Distinctiot between objective findings and opinions 



Comnunlaitioa and Disdoaure 



AO 


ruiauigs Clear ly presenica 




uuQcrwinoaPK; language 




Relative importance of findings 


A'X 


Acknowledgement ot assumptions 


AA 


L4muauons siaiea 


4 J. 


Complete description 


AA 


reeooacx to coninouiors 


47. 


Open disclosure 


48. 


Specifications of authorized to release dat j 


49. 


Database documentation 


50. 


Results available to users before decisions 


51. 


Anticipation of misinterpretation 


51 


Side effects reporting 


53. 


Distinction between findings and 




recommendations 


54. 


r.^nsideration of cost and effectiveness of 




corrective recommendations 


55. 


Distinction between advocates and 




evaluation role 
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Exhibit A-4. Topics covered in AERA/APA/NCM E Standards for Educational and Psychological Testing 



PaitL Tcdmksl Studards for Test 3. 
Constnictioo and EvaluaUoo 

L Validity 

Presenting evidence 
Assuming validity 
Supporting mterpretations 
Assessing responses to spedflc items 
Describing sample composition 
Reporting content-related evidence 
Noting experts* qualifications 
Measuring constructs 
Detailing a aitena-related validation 

study 

Making statistical adjustments 
Providing date/time information 
Conductmg differential prediction 

studies 

Comparing regression equations 
Classifying people into alternative 
treatment 

groups 

Recommending specific cut scores 

for 

decision making 
Accounting for bias in v^eights and 4. 

validity 

coefficients 

2. Reliability and errors of 
measurement 

Estimating relevant reliabilities, 
standard errors of 

measurement 

Describing reliability estimate 

methods 
Restricting range of reliability 

coefficients 

Administering alternate test forms 5. 
Basing coefficients on internal 

analysis 

Dealing with speeded tests 
Scoring tests 

Handling dififerent populations 
Reporting standard errors of 
measureinent 

Usi;^^: adoptive testing 
Making dichotomous decisions 

3. Test development acd revision 
Employing a scientific basts 
Designing test instruments 
Specifying domain defmitions 
Describing instructional materials 
Selecting test items 
Developing occupational interest 



Test development and revisioo 
(continued) 

Correlating item scores with criterion 
scores 

Using parameter estimates of item 
response curves 
Conducting adaptive tests 
Reviewing previous research 
Explaining test-taking strategies 
Investigating sources of variance 
Imposing strict time limits 
Studying sensitivity of test performance to 

improvement 
Measuring interest of personaliry 
Facilitating mterpretation of score report 
forms 

Prcparmg short forms 

Revismg tests 

Titling tests as "revised" 

Intending tests for research use only 

Presenting administration directions 

Colleaing structured behavior samples 

Clarifying test scoring procedures 

Judging test scores 

Scaling, norming, score comparability 
and equating 

Choosing scales for reporting scores 
Presentmg norms 
Reportmg norming studies 
Making norm-referenced assessments of 
groups 

Interchanging scores earned on different 

forms of a test 
Changing content specifications 
Using an anchor test design 
Continuing testing programs 

Test pui>U€atioo: technical manuals and 
user's guides 

Malting tedmical manuals available 
Amending test manuals 
Responding to requests for additional 

information 
Promoting a test 

Reporting relationsliips between test 

scores and criteria 
Recording responses by different methods 
Claiming a test is self-interpreting 
Automatin(^ test mterpretations 
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Exhflnt A-4. Topics covered in AERA/ APA/NCME Standards for Educational and Psychological Testing 
(continued) 



Part II. Profestlooal Studards for 
TctlUttt 

& GcBcrml priadpks of test use 

Evaluating written docuinentitio& 
Changing test conditions 
Using test for new purpose 
Portraying test relevance 
Alerting test users to probable 

unintended consequences 
Assuming responsibility 
Verifying effects of changes on 

procedures appropriateness 
Releasing test results to the sews mr.dia 
Presenting test score methods and 

rationale 
Evaluating spedal test takers 
Considering alternative explanations 
Identifying test takers needing further 

evaluation 
Interpreting test results 

1. Clinical tcstlnf 

Interpreting test data 
Appraising validity 
Distinguishing between diagnostic 
groups 

Determining if construct and 

assessment correspond 
Sharing results with clients 
Making validity evidence available 

8. EdncBtioaal testing and 

psych olofkai testing in the schools 
Instructing t«^ administraton 
Instructing users of test scores 
Differentiating between aptitude 
and achievement 
Certifying completion of a given 
edttcatioii level 
Developing tests 

Reporting certiScation test results 
Using tests for decision-making 
Demonstrating skills 
Describing rclatiomhtps between 

predicators and criterion 
Investigating potsitxlity of 

differential prediction 
Imptying existence of empirical 

evidence 
Makmg decisions affecting test 

takers 
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9. Test asc la couascllng 

Acquiring and reporting relevant 

iniformatioa 
Revievidng mterpretive materials 
Reviewing technical data 
Reviewing manuals 

Encouraging multiple valid assessments 

10. EmpioyBMBt testing 

Inferring criterion-related validity 
Making job classification decisions 
Detailing criterion relevance 
Defming content domain of interest 
Demonstrating test and job cooteot link 
Inferring test validity 

Supporting validity for personnel selection 
Explaining technical basis for cut score 

1 1. Profcsslooal and occupational licensure 
and certiflcatioa 

Defining content domain 
Elaborating construct interpretations of 
test 

Providing reliability estimates 
Informing test takers of their scores 

12. Program evaluatloa 

> Providing evidence of test validity, 

suitability 
Defming change of growth 
Calculating gain scores 
Aggregatingtest results 
Describing program effectiveness 
Merging student scores 
Evaluating service providers 
Using test results for fund allocation 

Part III. Standards for Particular 
Applicatioos 

13. Testing lingnlstk minorities 

Minimizing reliability and validity threats 
Describing linguistic modifications 
Providing test use and interpretation 

information 
Translating tests 

Testing for employment, licensing and 

certification 
Reporting test comparability 
Determining English language proficiency 



Exhibit A-4. Topics covered in AERA/APA/NCME Standards for Educational and Psychological Testing 
(cootinued) 

Part III. Suodards for PBiticular 
Applkatiotts (cooUoued) 

14. Testing people who have baodiaipplog 
coodltioos 

Modifying tests 
Validating teus 
Testing pilots 

Providing interpretive information 
Establishing time limits 
Investigating modification 

Interacting with test takers 
Assessing charaaeristics of individuals 

Part rV. Standards for Administrative 
rrocedurcs 

15. Test administratioo, scoring, and 
reporting 

Following standardised procedures 
Preparing test environment 
Assuring validity of test scores 
Modifying administration procedures 
Documenting procedures 
Correaing score reports 
Protecting security of test materials 
Checking accuracy of the scoring 
Retaining test data 
Providing appropriate bterpretation 
Maintaining test scores 

16. Protecting tlic rights of test 
talcers(lO) 

Obtaining informing consent 

Explaining test results 

Releasing test results 

Using scores to make decisions about 

individuals 
Protecting test data from improper 

disclosure 
Assigning mdividuals to cate^^nts 
Canceling test taker's scores 
Delaying score reports 
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Exhibit A-S. Topict covered in G AO Government Auditing Standards 



Standard 

GcMral Staadards (48) 

3-3 Staff collectively poaictt profideDcy 

3-11 Indepeadcoce (individual and organization free of impainnents) 
3-26 Exercise due professkxial care 

3- 43 Internal and external quality control 

Field Work for Floandal Audits (27) 

4- 4 Planning consider all levels 

4-6 Tasks of compliance Uws/regulations 

4-U Design to detect errors, irregularities, and illegal acts 

4-19 Record of work m the form of working papers maintained 

4- 23 Sufficient understanding of btemal structure to determine tests 

Reporting Standards for Flaandal AudlU (36) 

5- 3 Statement in accordance with AICPA standards 
5-5 Prepare written report on tests of compliance 
5- 17 Report on entit/s internal ^mtrols 

5-26 Written reports on the results of each financial related audit 
5-28 Reports state nature and reason of information omitted - confidential/i^vileged 
information 

5- 32 Distribute to appropriate persons/organizations 

Fidd Work for Standards Performance (73) 

6- 2 Work adequately plaimed 
6-24 Staff property supervised 

6-30 Assessment of complianoe of laws/regulations 

6-37 Design to detectable, illegal acts 

6-42 Be alert to situations indicative of abuse 

6-49 Assess internal oootrob 

6- S7 Suffideat evidence for conclusions 

RcpoctlBg S^s^^rds for fnforMoce AndlU (73) 

7- 2 Written re ports of results prepared 

7-10 Report inc2:?jcs objectives, scope and method 

7-17 Report fuD ditcmiion of findings 

7-21 Report cause of problems and recommendations 

7-27 Include statement oo standards 

7-29 Report identifies tignifiranr internal controls 

7-33 Report indudes incidence of noncompliance, abuse, and Ulegal acts 

7-48 Include description of noteworthy accomplishments 

7-50 Report issues warranting further audit work 

7-52 Include statement about priviky!4i/confidential information omitted 

7-55 Report must be complete, accurate, objective, convincing, and concise 

7-70 Report submitted to audited organization and others responsible for taking actions on 
audits 
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Exhibit A-6. Topics covered in National Center for Health Statistics (NCHS) Draft Cuideiines for 
Suitisiks and Informaaon on Effects of Environment on Health 



Guidelines for collcctioo 



Study plan 

Recommended data items 
Other data items 
Sample survey 

Interviews and questionnaires 



Guidelines for compHatioo 



Coding 
Editing 

Quality Control 
Nonresponse 



Guidelines for analysb 



Appropriateness 
Errors 

Analytic methods and statistical procedures 

Computer software 

Interpretation 



Guidciioes for publicaUoa 



Statistical design 
Measurement process 
Quality of data 
Technical review 
Timelines 

Guideliocs for dlstributloo 

Raw data 
Publications 

Appendix A: DnfUflf Qnestioannlre 

Appendix B: Pretest 

Appendix C: Privacy Act UmlUtioos 

Glossary 
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Exhibit A-7. Topics of the Energy Information Systems Standards Manual 



1. Orders 

3C Guidelines and Procedures for Mofel Documenution 

4C Guidelines and Procedures for the Preparation of Model Archival Packages 

IB Applications Support Software 

2. Intertgency Standards 

0 1 Federal Information Processing Standards (FIPS) 

02 Codes, Abbreviations, Acronyms 

03 OfGce of Management and Budget Standards 

3. DaU Systems Standards 

0 1 Contract Clauses for DaU Collection and Procedures 

02 Data Systems Development 

03 Data Systems Documentation 

04 Programing 

05 Frames Maintenance 

06 Formatting Survey Frames 

I Checklist (Explanation of the Data User's Manual Components) 

n Checklist (Explanation of Operations Manual Components) 

III Checklist (Explanation of Program Maintenance Manual Components) 

4. Data Collection Standards 

01 Survey Forms development 

02 Handling Nonresponse 

03 Respondent Contact Records (RCRs) 

04 Batch dEntry Procedures 

05 Edit Message Handling 

06 Performance Statistics 

IV Checklist (Forms Design) 

V Checklist (Collection Clearance Activities) 

5. Data PresenUtion Standards 

0 1 Publication of Energy Statistics 

02 Publication of Revisions 

03 Publication of Estimates 

04 Data Accuracy Presentation 

05 Statistical Graphs 

06 Nondisclosure of Company Identifiable Data in Aggregate Cells 

07 Rounding 

08 Freezing DaU Files 

09 Documenution of Public*Use Tapes/Diskettes 

Guidelines for ImplemenUtion of DaU Accuracy PresenUtion Standard 
Guidelines for ImplemenUtion of a Disclosure Avoidance Rule Sundard 
Guidelines on the Standare for Rounding 

VI Checklist (Survey Design Description) 
Vn Checklist (Text, Tables and Graphs) 

6. Glossaiy 
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Appendix 16 
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